Skip to content

Push expressions through layout plans - #9166

Draft
joseph-isaacs wants to merge 3 commits into
vortex-plan-rulesfrom
vortex-plan-optimizer
Draft

Push expressions through layout plans#9166
joseph-isaacs wants to merge 3 commits into
vortex-plan-rulesfrom
vortex-plan-optimizer

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Summary

  • replace the virtual Plan::optimize_expression hook with static typed parent-child rules
  • register rules for (ExpressionPlan, StructPlan), (ExpressionPlan, DictPlan), (ExpressionPlan, ChunkedPlan), and (ExpressionPlan, RowIdxPlan)
  • push independent struct-field expressions into each field while retaining cross-field work as a residual expression
  • push safe boolean expressions into dictionary values while retaining dictionary codes
  • optimize heterogeneous chunks independently while preserving global row-index semantics
  • partition mixed row-index/data expressions and combine their outputs with a residual expression
  • add complete before/after snapshots, including multi-field struct and row-index/struct composition

Stack

This draft is stacked on #9196 and targets vortex-plan-rules.

Example

Before optimization:

ExpressionPlan((#row_idx > 11) and (($.a > 5) and ($.b > 7)))
  RowIdxPlan
    StructPlan
      a: DictPlan
      b: FlatPlan

After optimization, the row-index predicate is evaluated by RowIdxValuesPlan, the field predicates are pushed into the corresponding struct children, and a residual expression combines the partition outputs. No synthetic struct-partition plan node is introduced.

Safety

  • dictionary pushdown requires boolean output, a root reference, strictness, and infallibility
  • nullable struct expressions remain above parent validity
  • chunk pushdown rejects expressions referencing global row indices
  • parent-reduction rules verify row-count and dtype preservation in debug builds
  • cross-field expressions that cannot be partitioned remain above the original struct plan

Checks

  • cargo test -p vortex-layout — 209 passed
  • cargo clippy -p vortex-layout --all-targets --all-features -- -D warnings
  • cargo +nightly fmt --all -- --check
  • git diff --check

@codspeed-hq

codspeed-hq Bot commented Aug 4, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 1830 untouched benchmarks
⏩ 43 skipped benchmarks1
🗄️ 12 archived benchmarks run2


Comparing vortex-plan-optimizer (bda6747) with vortex-plan-rules (ed09b04)3

Open in CodSpeed

Footnotes

  1. 43 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

  2. 12 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them.

  3. No successful run was found on vortex-plan-rules (1488672) during the generation of this report, so 8ebe5df was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@joseph-isaacs
joseph-isaacs marked this pull request as ready for review August 4, 2026 11:52
@joseph-isaacs
joseph-isaacs marked this pull request as draft August 4, 2026 11:52
@joseph-isaacs
joseph-isaacs force-pushed the vortex-plan-optimizer branch from 4dced78 to 7968c7e Compare August 4, 2026 16:48
@joseph-isaacs
joseph-isaacs force-pushed the vortex-plan-optimizer branch from 7968c7e to a9043d5 Compare August 5, 2026 14:35
@joseph-isaacs
joseph-isaacs changed the base branch from vortex-plan to vortex-plan-rules August 5, 2026 14:36
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
joseph-isaacs force-pushed the vortex-plan-optimizer branch from a9043d5 to bda6747 Compare August 6, 2026 09:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant